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A METHOD OF SELECTING TARGETS AND GENERATING 
FEEDBACK IN OBJECT TRACKING SYSTEMS 

BACKGROUND OF THE INVENTION 
5 The present invention relates to an improved 

video capture and display system. 

Object tracking systems are used for auto 
focusing, auto zoom, and other functions of video 
cameras. Likewise, object tracking systems are used for 

10 auto zoom, brightness adjustment, contrast adjustment, 
and highlighting functions of video display systems. 
Imaging devices in video cameras capture images by 
converting light into electrical signals through a 
photoelectric process. On the other hand, video display 

15 devices utilize electrical signals to control a light 

source producing an image. Object tracking systems use a 
number of known methods to process the electrical signals 
corresponding to an image and to extract a group of 
signals representing a designated target of the object 

20 tracking system. The group of signals representing the 
target object is relocated and "followed" in the 
succession of- images or frames making up a video 
sequence . 

To initiate the tracking process, a target must 
25 be initially identified for the object tracking system. 
Object tracking systems may automatically identify a 
target object based on an assumed usage model for the 
video capture or display device. For example, the object 
of interest will most likely to be located near the 
3 0 center of the image and may be the larger object in the 
vicinity. However, automatic target designation is 
complex and is problematic if the object of interest is 
not centrally located in the image, is one of a plurality 
of similar objects in the vicinity, or if there is little 
35 contrast between the object of interest and the 
surroundings . 



2 

User designation of the target object avoids 
some of the limitations and complexities of automatic 
target designation. Nishimura et al . , U.S. Patent 
No. 5,631,697, for VIDEO CAMERA CAPABLE OF AUTOMATIC 
5 TARGET TRACKING, describe a system in which the user 

designates the target by aligning an electronic marker 
generated in the viewfinder with the image of the 
intended target object and then depressing a switch to 
activate the target tracking processor. While less 

10 complex and more flexible than fully automatic target 
designation, the system provides only for user 
designation of the target object's location. 

To separate the group of signals representing 
the target object from the larger group of signals 

15 representing the image, target tracking systems utilize 
information about the size of the target object, as well 
as its location. Automatic or manual methods may be used 
to designate the target's size once its location has been 
manually designated. Automatic determination of the 

2 0 target object's size is complex and often unreliable, 

especially in crowded or low contrast images. To reduce 
the complexity of the tracking system and increase the 
reliability of target identification, the system may 
permit the user to designate the target's size. For 

25 example, the target object's size may be designated by 
drawing an "electronic" box bounding the target object. 
However, drawing a box around a moving target while 
operating a video camera is not easily accomplished and 
even locating a target in a viewfinder when there is 

30 little contrast between the target object and the 

surroundings or when the target is located at a distance 
may be difficult for a user. 

What is desired, therefore, is a method of 
designating a target object for an object tracking system 

35 that allows the user to easily locate the target object 
and reliably designate its location and size for the 
tracking system. Further, a method of notifying the user 
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of the continuing performance of the tracking system is 
desired . 

SUMMARY OF THE INVENTION 
5 The present invention overcomes the 

aforementioned drawbacks of the prior art by providing an 
object tracking system that includes initiating the 
object tracking system, magnifying the image, selecting 
an object of interest in an image, and designating the 

10 object as the target all while the object tracking system 
is activated. In this manner it is easier for the user 
of the object tracking system to select the target, 
especially if the target tends to be moving fast. 

In another aspect of the object tracking system 

15 an image is magnified and an object is designated as the 
target for tracking by the tracking system. In response 
to designating the object the scale of the image is 
automatically changed. In one embodiment, this permits 
the user to zoom in on the image, select the target more 

2 0 easily, and then automatically zoom out, which permits 

easier selection of the target. 

In yet another aspect of the prevent invention, 
the object tracking system monitors a level of confidence 
that the tracking system is properly tracking the target . 
25 In response to a change in the level of confidence the 
magnification of the image visible to the operator is 
altered. 

The foregoing and other objectives, features 
and advantages of the invention will be more readily 

3 0 understood upon consideration of the following detailed 

description of the invention, taken in conjunction with 
the accompanying drawings . 
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BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS 

FIG. 1 is a block diagram of a video camera 
employing the method of the present invention. 



4 



FIG. 2 is a flow chart of object tracking 
functionality. 

FIG. 3 is an exemplary viewfinder image of the 
video camera of FIG. 1. 
5 FIG. 4 is a magnified view of the exemplary 

viewfinder image of FIG. 2. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

Referring to FIG. 1, a video camera according 

10 to the present invention includes video capture and 
display systems. The video camera includes a 
lens unit 10, an imaging device 12 which may be a charge 
coupled device (CCD) or metal oxide semiconductor (MOS) 
array, an A/D (analog to digital) converter 14, a signal 

15 processor 16, a driver 18 to drive the imaging device 12, 
and a D/A (digital to analog) converter 20. The imaging 
device 12 includes a plurality of photoelectric elements 
arranged in a two-dimensional array on one surface. The 
light of an optical image is focused on the photoelectric 

20 elements of the imaging device 12 by the lens unit 10. 
Each photoelectric element converts the light from the 
image that is striking the element into an analog 
electrical signal. Periodically, the driver 18 connects 
elements of the imaging device 12 to the A/D converter 14 

25 causing the electrical signals produced by the elements 
of the imaging device 12 to be converted to digital 
signals which are read by the signal processor 16. The 
signal processor 16 converts the digital signals to video 
signals representing the luminance and chrominance of the 

30 light striking each element of the imaging device 12. 
The composite video signals are output to the D/A 
converter 2 0 for conversion to the analog signals of the 
camera's video output 22. The video output 2 2 may be 
transmitted or recorded and is displayed on a 

35 viewfinder 24 so that the user can visualize the images 
being transmitted or recorded by the camera. 



The video camera also includes a target 
tracking processor 26 that processes the output of the 
signal processor 16 to extract signals representing the 
target object from the signals representing the image in 
5 the succession of images or frames making up the video 

sequence. The video camera includes a zoom processor 28, 
a cursor generator 30, and a controller 32. The camera 
may also include a zooming motor 34 and an auto focus 
motor 3 6 driving the zooming and focusing mechanisms of 

10 the lens unit 10. As the lens 10 zooms in or out, the 
optical image is magnified on the imaging device 12 
producing a magnified image at the video output 22 and in 
the display of the viewfinder 24. It is to be understood 
that magnification may include the result of zooming in 

15 or zooming out. On the other hand, zooming may be 

accomplished by processing signals from a region of the 
imaging device 12. The zoom processor 28 may extract 
signals from a portion of the imaging device 12 and 
calculate zoom information from those signals to 

20 electronically magnify a portion of the image that is 
displayed in the viewfinder 24. 

Traditional video tracking systems include the 
user zooming in or zooming out until the object is 
clearly visible within the display and then the user 

25 initiates the tracking functionality. Thereafter, the 
user then selects the target to be tracked by the video 
tracking system. The present inventors came to the 
realization that while a workable technique for tracking 
targets, it is clumsy because by the time the user 

30 initiates the tracking functionality and then selects the 
target, the target may have already moved out of the 
field of view. This is especially true when the object 
takes up a significant portion of the display, such as 
when under extreme zoom. To overcome these limitations 

35 the present inventors have developed a new interface for 
a tracking system. Referring to FIGS. 2 and 3, first the 
tracking functionality is initialized by any appropriate 



technique. Next, the target is located, sized, and 
positioned on an electronic viewfinder 24 for accurately 
selecting and subsequent tracking. A cursor 42 is 
generated by the cursor generator 3 0 and is superimposed 
5 on the image in the electronic viewfinder 24 . The 

cursor 4 0 may be in a fixed location in the viewfinder 24 
{e.g., centered) and positioned over the image of the 
target object by movement of the camera which effectively 
moves the viewfinder image behind the cursor 42 . On the 

10 other hand, the cursor may be movable and electronically 
positioned in the viewfinder image by operation of a 
joystick, touch pad or other pointing device 38 (refer to 
FIG. 1) . A target designator switch 39 facilitates 
signaling the controller 32 that a target is being 

15 designated for tracking. The target designator switch 3 9 
and the pointing device 3 8 may be combined into a single 
control element permitting the user to conveniently 
position a movable cursor 42 and designate the target. A 
movable cursor 42 permits the user to designate target 

2 0 objects that are not centered in the image. To initiate 

tracking, the user aligns the cursor 42 to superimpose 
the cursor 42 on the target object and actuates the 
target designator switch 39. Preferably, the target 
designator switch 39 and the pointing device 3 8 are 
25 combined by touch sensitive viewfinder 24. In this 

manner, the user may simply target and select the target 
to be tracked by a single action. The physical activity 
used on the video camera for targeting and selecting 
(e.g., buttons and controls) are preferably different 

3 0 than that used for zooming, which permits the targeting 

and selecting functions to be placed into a "ready" state. 
In the case of the touch screen viewfinder 24, while 
touching the viewfinder 24 may be used for many different 
functions, after initialing the tracking functionality 
35 the touch screen viewfinder 's next function is 

automatically set to targeting and selecting of the 
target. Accordingly, after zooming a single action of 



touching the viewfinder 24 screen will perform targeting 
and selecting in a fast and efficient manner before the 
target, especially if moving fast, is no longer displayed 
on the viewfinder 24. 
5 In one embodiment of the present invention, 

actuation of the target designator switch 39 causes the 
controller 36 to signal the zoom processor 28 to begin 
zooming. The zoom processor 2 8 performs the zoom 
calculation or actuates the zoom motor 34 causing the 

10 zoom lens unit 34 to zoom in, magnifying the image in the 
viewfinder 24. Referring to FIG. 4, the image in the 
viewfinder 24 is magnified until the target object 44 is 
easy to view and designate. The user then designates the 
target object by actuating the target designator switch 

15 3 9 for a second time. This may likewise be performed by 
a touch sensitive viewfinder 24 where after initiating 
tracking functionality the user touches the screen to 
start zooming and touches the screen a second time to 
designate the target. The first zooming may be 

20 automatic, if desired. Designation of the target causes 
the target tracking processor 26 to extract the signals 
representing a target designated at the location of the 
cursor (or touch on the screen) from the signals of the 
image in a scan of the imaging device 12 . A target may 

25 be identified in any manner. The target tracking 
processor 26 then "follows" this group signals 
representing the target object in subsequent scans of the 
imaging device 12 to track the target as the video 
sequence progresses. 

3 0 In another embodiment of the present invention, 

actuation of the target designator switch 39 causes the 
controller 36 to signal the target tracking processor to 
initiate tracking and the zoom processor 28 to begin 
zooming in. When the image of the tentative target is 

35 easy to view in the viewfinder 24 the user designates the 
target object a second time by actuating the target 
designator switch 39. The target object is tentatively 
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identified for the target tracking processor 26 by the 
first actuation of the target designator switch 39 but 
the information about the size and location of the target 
is refined and verified in the magnified image where the 
5 target object is more visible. 

The zoom processor 2 8 may then zoom out 
electronically or by actuating the zoom motor 34 to 
return the zoom to the original setting or some 
intermediate setting established by the user. The 

10 present inventors realized that zooming in on the target 
object would make it easier to locate the target object 
if it is at a distance from the camera or in low contrast 
or crowded surroundings. Further, automatically zooming 
the image provides a convenient method of establishing 

15 the target. This may likewise be performed with a touch 
sensitive display. 

The algorithm applied by the target tracking 
processor 2 6 may include a measurement of the level of 
confidence that the target is being correctly tracked. 

2 0 If the level of confidence should decrease, the tracking 

processor 26 may cause the zoom processor 2 8 to begin 
zooming in. The changing magnification of the image in 
the viewfinder 24 notifies the user that the confidence 
level of the algorithm is low. Increasing the 
25 magnification of the image may cause the algorithm's 

confidence level to increase. On the other hand, if the 
algorithm cannot automatically reacquire the target 
object, increasing the magnification of the image will 
facilitate relocation of the target object by the user 

3 0 who may re -designate the target for the tracking system 

by actuating the target designator switch 39. 

While tracking a target periodically the target 
may be obscured or otherwise not visible on the 
viewfinder 24 for a period of time. This may occur, for 
3 5 example, when another object comes in between the target 
and the camera for a period of time. When the target is 
not detected by the target tracking processor, the system 



may automatically switch into an alternative global 
tracking motion mode to predict where the object should 
be. In this manner, when the target is no longer 
obscured or otherwise not visible on the viewfinder 24 
5 the target tracking processor will have an increased 
change of identifying the target to continue tracking. 
In this manner, the system is less likely to loose track 
of the target . 

When the confidence of the system is 

10 sufficiently low (or losing the target totally) the 
present inventors came to the realization that most 
likely the target is no longer visible within the 
viewfinder 24 , Accordingly, in such an event the system 
may automatically zoom out {or in) to present the viewer 

15 with an image having a greater (or lesser) field of view. 
In most instances, the target will be visible to the user 
within the new field of view image. The user then may 
select the target within this field of view in the manner 
described above, as desired. Alternatively, after the 

2 0 zooming in response to the confidence level the system 
may exit the tracking functionality. 

Yet in another aspect of the present invention, 
the present inventors came to the realization that 
control over the amount of magnification provided by the 

2 5 system for the selection of the target should operate in 

an intuitive and efficient manner. The intuitive system 
developed by the present inventors includes the user 
selecting a target object for tracking, in any manner, 
such as those previously described. Preferably, after 

3 0 selecting the target object the tracking system tracks 

the target object. In the event that the user triggers 
another target object selection within a predetermined 
time period, then the system will cease object tracking, 
if object tracking was already initiated, and change the 
3 5 magnification of the image on the display. The same or a 
different object may be selected for the another target 
object selection. For example, the magnification for the 
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target objection selection may be performed by, for 
example, clicking a mouse twice (or multiple times) or 
touching a display twice (or multiple times) . After 
changing the magnification of the image, the system then 
5 permits the user to select a target object for tracking. 
Again, if the user triggers another target object 
selection within a predetermined time period, then the 
system will cease object tracking, if object tracking was 
already initiated, and change the magnification of the 

10 image on the display (such as increase magnification) . 

This process is repeated until the user does not trigger 
another target object selection within a predetermined 
time period. This permits magnification of the image 
until the target can be easily selected for tracking. 

15 The target selected for tracking is preferably the last 
selected object. After tracking is initiated, and 
preferably the predetermined time period has elapsed, the 
system may return to a previous magnification, if 
desired . 

2 0 While the tracking system has been described in 

relation to a video camera, the teachings are equally 
applicable to selection and tracking when viewing video, 
from any source, such as for example, a video cassette 
recorder, digital tape, analog tape, compact disc, and 

25 digital video disc. 

All the references cited herein are 
incorporated by reference. 

The terms and expressions that have been 
employed in the foregoing specification are used as terms 

30 of description and not of limitation, and there is no 

intention, in the use of such terms and expressions, of 
excluding equivalents of the features shown and described 
or portions thereof, it being recognized that the scope 
of the invention is defined and limited only by the 

35 claims that follow. 
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CLAIMS 



1 . A method of tracking a target for an 
object tracking system comprising the steps of: 

(a) initiating said object tracking system; 

(b) magnifying said image while said object 
tracking system is activated; 

(c) selecting an object of interest in an 
image while said object tracking system is 
activated; and 

(d) designating said object as said target of 
said tracking system while said object 
tracking system is activated. 



15 2 . The method 

is magnified by adjustmeni 

3 . The method 
is magnified by adjusting 
20 representing, at least, a 



of claim 1 wherein said image 
: of an optical lens. 

of claim 1 wherein said image 
an electrical signal 
part of said image . 



4 . The method of claim 1 wherein said 
magnification is an automatic result of said initiating 
said object tracking system. 

25 

5. The method of claim 1, further comprising 
the step of automatically changing the scale of said 
image following designation of said object as said 
target . 

30 

6. The method of claim 1 wherein said object 
of interest is selected by the steps of: 

(a) moving a cursor to superimpose said cursor 
on said object of interest in said image; 
3 5 and 
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(b) signaling said tracking system that said 
cursor is superimposed on said object of 
interest . 

5 7. The method of claim 1 wherein said 

designating is using a touch sensitive display. 

8 . The method of claim 1 wherein said 
selecting said object of interest and said designating 

10 said object uses a different control mechanism than said 
magnifying said image. 

9 . The method of claim 1 wherein said 
selecting and said designating are performed 

15 simultaneously by touching a touch sensitive display. 

10. The method of claim 9 wherein in response 
to initiating said object tracking system, said touch 
sensitive display is set to simultaneously said select 

2 0 and said designate upon the next touch of said touch 
sensitive display. 

11. The method of claim 6 wherein said image 
is magnified by adjustment of an optical lens. 

25 

12 . The method of claim 6 wherein said image 
is magnified by adjusting an electrical signal 
representing, at least, a part of said image. 

30 13. The method of claim 6 wherein said 

magnification is an automatic result of said initiating 
said object tracking system. 

14. The method of claim 6 further comprising 
35 the step of automatically changing the scale of said 
image following designation of said object as said 
target . 



15. The method of claim 1 wherein said object 
of interest is selected by the steps of: 

(a) moving said image to superimpose an image 
of a cursor on said object of interest; 
and 

(b) signaling said tracking system that said 
cursor is superimposed on said object of 
interest . 

16. The method of claim 15 wherein in response 
to initiating said object tracking system, said touch 
sensitive display is set to simultaneously said select 
and said designate upon the next touch of said touch 
sensitive display. 

17. The method of claim 15 wherein said image 
is magnified by adjustment of an optical lens. 

18. The method of claim 15 wherein said image 
is magnified by adjusting an electrical signal 
representing, at least, a part of said image. 

19. The method of claim 15 wherein said 
magnification is an automatic result of said initiating 
said object tracking system. 

20. The method of claim 15 further comprising 
the stop of automatically changing the scale of said 
image following designation of said object as said 
target . 

21. A method of selecting a target for an 
object tracking system comprising the steps of: 

(a) magnifying an image; 

(b) designating an object in said image as a 
target for tracking by said tracking 
system; and 
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(c) in response to said designating of said 
object as said target, automatically 
changing the scale of said image. 

5 22. The method of claim 21 wherein said image 

is magnified by adjustment of an optical lens. 

23. The method of claim 21 wherein said image 
is magnified by adjusting electrical signals 

10 representing, at least, a part of said image. 

24. The method of claim 21 wherein said 
magnification is an automatic result of initiating said 
object tracking system. 

15 

25. The method of claim 21 wherein said first 
and second designating of said object as said target 
comprises the steps of: 

(a) moving a cursor to superimpose said cursor 

2 0 on said object in said image; and 

(b) signaling said tracking system that said 
cursor is superimposed on said object. 

26. The method of claim 21 wherein said first 
25 and second designating of said object as said target 

comprises the steps of: 

(a) moving said image to superimpose a cursor 
on said object; and 

(b) signaling said tracking system that said 

3 0 cursor is superimposed on said object. 

27. method of advising an operator of the 
performance of an object tracking system comprising the 
steps of: 

35 (a) monitoring a level of confidence that said 

tracking system is tracking a target; and 
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(b) altering magnification of an image visible 
to said operator in response to a change 
in said level of confidence. 



28. The method of claim 27 wherein said 
magnification is changed as said level of confidence 
decreases . 



29. The method of claim 27 wherein said 
10 magnification is decreased if said object tracking system 
loses track of said target. 



30. A method of selecting a target for an 
object tracking system comprising the steps of: 
15 (a) a first designating of an object in said 

image as a target for tracking by said 
tracking system; 

(b) magnifying said image if a second 
designating of at least one of said object 

2 0 and another object in said image is 

performed within a predetermined time 
period; 

(c) repeating steps (a) and (b) until said 
second designating is not said performed, 

2 5 and in response tracking said object. 



31. The method of claim 30 wherein said first 
designating of said object comprises the steps of: 

(a) moving a cursor to superimpose said cursor 
3 0 on said object in said image; and 

(b) signaling said tracking system that said 
cursor is superimposed on said object. 



32. The method of claim 30 wherein further 
35 comprising the step of changing the magnification of said 
image in response to said tracking of said object. 



16 



A METHOD OF SELECTING TARGETS AND GENERATING 
FEEDBACK IN OBJECT TRACKING SYSTEMS 

ABSTRACT OF THE DISCLOSURE 
5 A system initiates an object tracking system, 

magnifies the image, selects an object of interest in an 
image, and designates the object as the target all while 
the object tracking system is activated. In this manner 
it is easier for the user of the object tracking system 
10 to select the target, especially if the target tends to 
be moving fast . 
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97258, telephone number 503-227-5631, my attorneys, jointly 
and individually, to prosecute this application and to 
transact all business in the Patent and Trademark Office 
connected therewith. 

We hereby declare that all statements made herein of our own 
knowledge are true and that all statements made on information 
and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false 
statements and the like so made are punishable by fine or 
imprisonment, or both, under Section 1001 of Title 18 of the 
United States Code and that such willful false statements may 
jeopardize the validity of the application or any patent 
issued thereon. 




Q^rge Borden IV 
^^ancouver, Washington 
United States of America 
11301 SE 10^^ St . , #191 
Vancouver, WA 98664 



Full name of sole inventor 

Residence 

Citizenship 

Post Office Address 




Residence 
Citizenship 
Post Office Address 



United States of America 
27119 NE Bradford Road 
Vancouver, WA 98682 



-2- 



Dated: /-~J2-0O j^^^«^X-^^i>CllXj , Q-^L^ 

Full name of sole inventor Richard Qian . _ , 

Residence Camas, ^Oregon ^vJf\s'^'W'^*^*'' 

Citizenship China ^ 

Post Office Address 2 844 NW 44^"^ Avenue 

Camas, WA 98607-8318 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
CERTIFICATE UNDER 37 CFR ^3 . 73 (b) 



Applicant : Borden et al . 

Application No: Filed: Concurr ently herewith 

Title : METHOD OF SELECTING TARGETS AND GENERATING FEEDBACK IN OBJECT 
TRACKING SYSTEMS 

Sharp Laboratories of America, Inc. (Assignee) a Washington corporation 

(Type of Assignee, e.g., corporation, partnership, university, government 
agency, etc .) certifies that it is the assignee of the entire right, title and 
interest in the patent application identified above by virtue of either: 

A. [ ] An assignment from the inventor (s) of the patent application 

identified above. The assignment was recorded in the Patent and 

Trademark Office at Reel , Frame , or for which a 

copy thereof is attached. 



A chain of title from the inventor (s), of the patent applicatio 
identified above, to the current assignee as shown below: 



1. From: To: 

The document was recorded in the Patent and Trademark Office at Reel 
, Frame , or for which a copy is attached. 



The document was recorded in the Patent and Trademark Office at Reel 
Frame , or for which a copy is attached. 

al documents in the chain of title are listed on a supplemental 

[x] Copies of assignments or other documents in the chain of title 

are attached. (Where one or more of the documents is 
unrecorded. ) 

The undersigned has reviewed all the documents in the chain of title of 
the patent application identified above and, to the best of undersigned's 
knowledge and belief, title is in the assignee identified above. 

The undersigned (whose title is supplied below) is empowered to sign 
this Certificate on behalf of the assignee. 

I hereby declare that all statements made herein of my own knowledge are 
true, and that all statements made on information and belief are believed to 
be true; and further, that these statements are made with the knowledge that 
willful false statements, and the like so made, are punishable by fine or 
imprisonment, or both, under Section 1001, Title 18 of the United States Code, 
and that such willful false statements may jeopardize the validity of the 
application or any patent issuing thereon. 

Date : February 16, 2000 ^ — 



Attorney for the Assignee (a copy of the Powe r of Attorney 



Signature: ^. 

Chernoff, Vilhauer, McClung & Stenzel, LLP 

1600 ODS Tower 

602 SW Second Avenue 

Portland, Oregon 97204 

Tel: (503) 227-5631 



